{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 7 - Linear regression continued\n",
    "\n",
    "We will continued learning about linear regression by predicting health insurance prices.\n",
    "\n",
    "First download the dataset from GitHub: [https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/insurance.csv](https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/insurance.csv)\n",
    "\n",
    "In this data, each row represents an insurance policy and the 7 columns contain the following information about it:\n",
    "- age: age of policy holder\n",
    "- sex: sex of policy holder\n",
    "- bmi: boday mass index (bmi) of policy holder.  bmi is a (sometimes unreliable) measurement of body fat in adults\n",
    "- children: number of children (dependents) on the policy\n",
    "- smoker: whether the policy holder is a smoker\n",
    "- region: region of the country the policy holder lives in\n",
    "- charges: price for insurance policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import statsmodels.formula.api as smf\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the CSV file into a dataframe and display it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis\n",
    "\n",
    "To get a feel for the data, let's do some quick exploratory data analysis.\n",
    "\n",
    "What's the histogram of the bmi column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What distribution does the bmi data have?\n",
    "\n",
    "Plot scatter plots of all pairs of quantitative variables (hint:  use the Seaborn function `pairplot` from Lab 1 to plot them all at once)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do any of the variables have a linear relationship with the charges?\n",
    "\n",
    "Use Seaborn to make a scatter plot with bmi on the x axis, charges on the y axis, and colored by whether the person is a smoker or not (see Lab 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which appears to have the larger effect on the charge:  the policy holder's bmi or whether they are a smoker?\n",
    "\n",
    "Next use Seaborn to make a scatter plot with age on the x axis and charges on the y axis, colored by whether the person is a smoker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the plot?\n",
    "\n",
    "## Linear regression\n",
    "\n",
    "Perform linear regression to predict the insurance charge, with age as the independent variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the equation for the linear model?\n",
    "\n",
    "How much does this model predict your insurance will increase next year when you are 1 year older?\n",
    "\n",
    "How much would a 25 year old pay?  We can predict this using our model.  First we make a new DataFrame with the age with want to make the prediction for. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_data = pd.DataFrame({'age' :[25]})\n",
    "new_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we make the prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm.predict(new_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about if you are 30, 40, or 50 years old?  We can compute the predicted charges of all of these ages at once by making a data frame containing all three ages: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "three_ages = pd.DataFrame({'age': [30,40,50]})\n",
    "three_ages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make the prediction using this new dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Use the Seaborn package to plot a scatter plot of age vs charges with the regression line on it (see Lab 6):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the scatter plot?\n",
    "\n",
    "Let's see if this shows up in the plots of the residuals.  Plot the histogram of the residuals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does this look like a normal distribution?\n",
    "\n",
    "Let's also plot the fitted values (y) against the actual charges (x):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could plot the ages (x) vs. the residuals (y):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly age does not provide the whole picture.  In fact, the R-squared value in the summary (top right corner) is the proportion of variance in the charges that is explained by this model.  Right now this is about 9% which is not good....\n",
    "\n",
    "However we see that the p-values (the P > |t|) column in the summary is very close to 0.  The p-value is the probability that that coefficient is 0, so there is a linear effect of age on insurance prices.\n",
    "\n",
    "Let's add the other quantitative columns as independent variables to see if we can get a better fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm2 = smf.ols('charges ~ age + bmi + children', data = insurance).fit()\n",
    "lm2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the equation of this linear model?\n",
    "\n",
    "Has the R-squared value improved?\n",
    "\n",
    "Looking at the p-values, could any of the coefficients be 0? \n",
    "\n",
    "Now let's plot the residuals.  First, plot a histogram of the residuals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?  Are the residuals normal?\n",
    "\n",
    "Next, let's plot the actual charges (x) vs the predicted charges (y):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did adding bmi and children improve the linear model?\n",
    "\n",
    "Let's add the remaining columns.  Sex, smoker, and region are all categorical variables.  But there is a way to make them into quantitative data using *dummy variables*.\n",
    "\n",
    "For example, consider the sex column.  There are two categories in it: female and male.  We will encode this using one dummy variable that will be 1 if the sex is male and 2 if the sex is female."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "insurance_new = pd.get_dummies(insurance, columns = [\"sex\"], drop_first = True)\n",
    "insurance_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the other qualitative columns into dummy variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "insurance_new = pd.get_dummies(insurance_new, columns = [\"smoker\", \"region\"], drop_first = True)\n",
    "insurance_new.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How was the region column, which had 4 categories, turned into dummy variables?\n",
    "\n",
    "Now let's make a linear regression model using all these columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Has the R-squared value improved?\n",
    "\n",
    "Looking at the p-values, could any of the coefficients be 0?  Next class will we learn how to decide which independent variables to include in your linear model.\n",
    "\n",
    "Now let's plot the residuals.  First, plot a histogram of the residuals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?  Are the residuals normal?\n",
    "\n",
    "Next, let's plot the actual charges (x) vs the predicted charges (y):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?  Has the model improved?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}